EM Algorithm and its Application

The expectation-maximization (EM) algorithm aims to nd the maximum of a log-likelihood function, by alternating between conditional expectation (E) step and maximization (M) step. This survey rst introduces the general structure of the EM algorithm and the convergence guarantee. Then Gaussian Mixture Model (GMM) are employed to demonstrate how EM algorithm could be applied under Maximum-Likelihood (ML) criteria. By introducing proper priors, the object function to be maximized under Maximize a Posteriori (MAP) creteria could also t in the log form and be evaluted via EM algorithm. At last, some extentions and improvement of EM algorithm are brie y stated. 1 Intuition for the EM Algorithm Traditionally, EM algorithm is an optimization method for numerical evalutation of maximum-likelihood. It aims to nd maximum likelihood estimates of parameters in statistical models, where the model depends on unobserved variables. We de ne some notations rst. θ : the model parameters y : the observed (visible) variables x : the unobserved (hidden/latent/missing) variables p (x, y | θ) : the joint distribution of complete data L (θ) = log (p (y | θ)): the object function to be maximized There are two main applications of the EM algorithm[10]. The rst application of EM algorithm occurs when the data indeed has missing values, due to limitations of the observation process. The second occurs when optimizing the likellihood function is analytically intractable. However, by assuming the existence of certain hiddden or latent variables, the problem could be simpli ed and easier to handle. The intuition behind EM is to alternate between estimating the hidden variables and the unknown parameters. Let's rst examine a simple example and see how EM algorithm works[17].